Season Image Classifier


This Season Image Classifier was inspired by a Day and Night Image Classifier project as part of Udacity's Computer Vision Nanodegree training.

Dataset

The dataset consists of [280] RGB color images split into two directories and four categories each (spring, summer, fall, and winter):

img_train: 200 images for training (50 of each season)
img_test: 80 images for testing (20 of each season)

The initial goal of this project is to build a classifier that can accurately label these images based on the season in which they were taken. This relies on finding distinguishing features between the different types of images.

Note: I'm trying to collect images from the AMOS dataset (Archive of Many Outdoor Scenes), but at the moment, I'm using CC0 images collected from Pexels.

Import resources


1. Load the datasets

First, we load in the training images and store them in a variable called, IMG_LIST.

Visualize the input images


2. Pre-process the data

Standardize input

We need to resize each image to the desired input size: (hxw)

Standardize output

There are three ways we can categorize images:

  1. Categorical (winter, spring, summer, fall)
  2. Integer encoding (0 = winter, 1 = spring, 2 = summer, 3 = fall)
  3. One-hot encoding (winter = [1,0,0,0], spring = [0,1,0,0], summer = [0,0,1,0], fall = [0,0,0,1])

Standardize IMG_LIST

Construct a STD_LIST of input images and output labels.

Visualize the standardized data

Multiple samples

I want to check multiple samples to try and notice some patterns. So, let's display 4 rows of images from each season. Column 1 will contain winter images, Column 2 is spring, Column 3 is summer, and Column 4 is fall images. And let's turn this into a function so we can do it again later with another set of images.


3. Feature Extraction: Average Brightness

To extract the average brightness of an image, we use the HSV colorspace, specifically the V (value) channel as the measure of brightness.

  1. Convert to HSV colorspace
  2. Add up the pixel values in the V channel
  3. Divide that sum by the area of the image to get the average brightness of the image

RGB to HSV conversion

First, let's visualize the RGB channels, then we can convert to HSV.

Find average brightness of the V channel

Visualize the data (average brightness)


4. Feature Extraction: Colors

Actually, we need to consider multiple color spaces in order to detect patterns in these images so that we can attempt to classify them correctly. So, let's take a look at some of these color spaces, what they mean, and how they are used.

Color space CV Code Channel 1 Channel 2 Channel 3 Notes
XYZ 33 X = mix of three CIE RGB curves chosen to be non-negative Y = Luminance Z = Quasi-equal to blue (CIE 1931) Widely used in scientific work
YCrCb 37 Y = Luminance Cr = Red-difference chroma Cb = Blue-difference chroma similar to YUV
HSV 41 H = Hue (color value) S = Saturation V = Value (brightness, max value = intense color, min value = black) similar to HSL
Lab 44 L = Perceptual Lightness a = color axis relative to green-red b = color axis relative to blue-yellow similar to Luv
Luv 51 L = Luminance / lightness u = green/red axes colors v = blue/yellow axes colors (CIE 1976) Widely used in computer graphics dealing with colored lights (also pedestrian, building, etc detection) similar to Lab
HLS 53 H = Hue (color value) S = Saturation L = Lightness (max value = white) similar to HSV
YUV 83 Y = Luma component U = Blue projection chroma V = Red projection chroma similar to YCrCb

RGB to other color spaces conversion

Let's create a function to handle multiple conversions and print images from each season with conversions. Then, we'll compare the differences between some of the more similar color spaces in order to choose the ones we want to focus on.

HSV vs HSL

YCrCb vs YUV

Lab vs Luv

XYZ vs Luv

XYZ vs RGB

Check a color space for each season

Color spaces:

Color space choices:

  1. HSV : particularly for H = Hue, and V = Value (brightness)
  2. Lab : a = green/red axes, b = blue/yellow axes
  3. YUV : U = blue, V = red (opposite Lab)
  4. RGB : R = red, G = green (handled with a in Lab), B = blue

0. Winter samples

1. Spring samples

2. Summer samples

3. Fall samples

Average each channel for each season for every image

Now that we've done a lot of processing, chosen a few color channels to focus on, and tried to search out a few patterns, let's find the average value for EVERY channel over EVERY image in each category. Basically, in order to do this, we need to do X things, and we can use some previously defined functions to do so:

  1. Determine which season we are "in"
  2. Starting at the beginning of that season's images, loop through the entire 50 image set
  3. Read in an image
  4. Convert it to the appropriate color space (there are 4)
  5. Split image into 3 color channels
  6. Get the average for each channel (get_channel_avg(channel))
  7. Keep a running total of the average for each channel
  8. Return the averages for each channel (and print them)

Find each channel's standard deviation for each season

Finding the standard deviation may help us to determine which types of images have large or small variation in their average data values of their color channels. This may help us to single out one or more characteristics that make it easier to classify certain types of images more quickly.

For example, if Fall images always have muted colors within a certain range, it will be easier to pick those out from the start.

If Winter images almost always have color values along the RGB spectrum that are have little variance (i.e. the images are mostly white and black, with little bright colors that stand out), then it will be easier to pick those out and label them quickly as well.

Once we have one or two features that are easy to find, and therefore classify, we will have less features we need to look for later, by the process of elimination.

Unique characteristics of our images (by season)

Understanding the unique features of each season (as determined by our previous calculations using various color channels) will help us build our Classifier in the next section.


5. Classification

Now, let's turn our feature extractors into classifiers that take in a standardized image and return a predicted_lbl for that image. The estimate_lbl() function should return a value: 0, 1, 2, or 3 (spring, summer, fall, winter).

Testing the Classifier

Now, let's test this classification algorithm against our Test Data that we set aside at the beginning of the notebook. Because our classifier currently is pretty simple, we shouldn't expect 100% accuracy, but let's shoot for between 75-85% accuracy.

  1. Standardize the Test data like we did the training data
  2. Shuffle it (to ensure order does not play a role in testing accuracy)
  3. Visualize the Test data
  4. Determine the Accuracy

6. Determining Accuracy and Visualizing Error

Determine the Accuracy

Compare the output of our Classification algorithm (our "model") with the true labels and determine the accuracy.

Then, store all misclassified images, their predicted labels, and their true labels, in a list called misclassified.

Visualize misclassified images

It's important to take a look at some of the images our Classifier misclassified in order to determine WHY they were misclassified. What did the algorithm calculate that doesn't fit in this image? Are there any weaknesses in the classification algorithm?


7. Improving the algorithm and Future Research

Now, it may be time to improve the algorithm. What can be improved upon?

Future Research

This project illustrated how various color channels, color conversions, and pattern recognition on the part of a human programmer may be used to help develop an algorithm to classify images by season. However, there are various weaknesses inherent in this approach. Namely, that only color channels, such as Hue, Lightness, Red, Blue, and various combinations are used to classify images. This project does not take into account any sort of edge detection, or pixel gradient variation to help it classify images.

For future research, various types of Neural Networks, such as CNNs and RNNs will be explored. Using a combination of data augmentation, convolutional layers, max pooling layers, batch normalization, dropout, and softmax layers, we will attempt to create better models that are able to classify images more accurately and within a faster time frame.

Additionally, it would be interesting to compare the accuracy rate and speed of various Neural Network architectures.